Papers with Language modeling
Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent studies suggest the use of general language models for improving natural language processing tasks. |
| Approach: | They propose a method that leverages the second phase to its fullest by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. |
| Outcome: | The proposed method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. |
Modeling Code-Switch Languages Using Bilingual Parallel Corpus (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for bilingual language modeling are limited due to lack of training data and syntactic structure. |
| Approach: | They propose a bilingual attention language model that performs language modeling objective with a quasi-translation objective to model the monolingual and cross-lingual sequential dependency. |
| Outcome: | The proposed model reduces the perplexity of 20.5% over the best-reported model. |
Attention-based Contextual Language Model Adaptation for Speech Recognition (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing language models do not incorporate utterance level contextual information . however, for some domains like voice assistants, additional context provides a rich input signal . |
| Approach: | They propose a method for training neural speech recognition models on text and contextual data. |
| Outcome: | The proposed model reduces perplexity by 7.0% relative over a standard LM . it also improves perxicity by 2.8% relative to a state-of-the-art model for contextual LM. |
Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training (D19-1)
Copied to clipboard
| Challenge: | Code-switching (CS) is a linguistic phenomenon defined as "the alternation of two languages within a single discourse, sentence or constituent." |
| Approach: | They propose an ASR-motivated evaluation setup which is decoupled from an ASL system and the choice of vocabulary . they propose a discriminative training approach which works better than generative language modeling . |
| Outcome: | The proposed evaluation setup is better than generative language modeling, the authors show . the proposed setup is decoupled from an ASR system and the choice of vocabulary . |
Effects of sub-word segmentation on performance of transformer language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models are a fundamental task in natural language processing, but few studies focus on the effect of sub-word segmentation on the performance of models. |
| Approach: | They compare GPT and BERT models trained with statistical segmentation algorithm BPE to unsupervised morphological segmentation algorithms Morfessor and StateMorph. |
| Outcome: | The proposed model trains for several languages and compares them with two unsupervised morphological segmentation algorithms. |
A Measure-Theoretic Characterization of Tight Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Language modeling is a core task in natural language processing. |
| Approach: | They propose to characterize leakage onto the set of infinite sequences by a measure-theoretic approach. |
| Outcome: | The proposed language model families are tight, meaning they will not leak . the proposed language models are based on the 'sequence leakage' hypothesis . |